[SPARK-14396] [SQL] Throw Exceptions for DDLs of Partitioned Views by gatorsmile · Pull Request #12169 · apache/spark

gatorsmile · 2016-04-05T05:48:29Z

What changes were proposed in this pull request?

Because the concept of partitioning is associated with physical tables, we disable all the supports of partitioned views, which are defined in the following three commands in Hive DDL Manual:

ALTER VIEW view DROP [IF EXISTS] PARTITION spec1[, PARTITION spec2, ...];

ALTER VIEW view ADD [IF NOT EXISTS] PARTITION spec;

CREATE VIEW [IF NOT EXISTS] [db_name.]view_name [(column_name [COMMENT column_comment], ...) ]
  [COMMENT view_comment]
  [TBLPROPERTIES (property_name = property_value, ...)]
  AS SELECT ...;

An exception is thrown when users issue any of these three DDL commands.

How was this patch tested?

Added test cases for parsing create view and changed the existing test cases to verify if the exceptions are thrown.

hvanhovell · 2016-04-05T06:52:42Z

sql/catalyst/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBase.g4

    | ALTER TABLE tableIdentifier
        SET SKEWED LOCATION skewedLocationList                         #setTableSkewLocations
-    | ALTER TABLE tableIdentifier ADD (IF NOT EXISTS)?
+    | ALTER kind=TABLE tableIdentifier ADD (IF NOT EXISTS)?


Minor: we don't really need kind. We can also just check for the existance of the VIEW keyword, e.g.: ctx.VIEW != null.

Thanks! Will do.

SparkQA · 2016-04-05T07:31:57Z

Test build #54959 has finished for PR 12169 at commit b6c1601.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

# Conflicts: # sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveDDLCommandSuite.scala

SparkQA · 2016-04-05T15:21:14Z

Test build #54983 has finished for PR 12169 at commit 140f859.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

gatorsmile · 2016-04-05T16:55:49Z

Thanks for the review! @hvanhovell

Also cc @yhuai @andrewor14

andrewor14 · 2016-04-05T19:30:26Z

sql/core/src/main/scala/org/apache/spark/sql/execution/SparkSqlParser.scala

   */
  override def visitAddTablePartition(
      ctx: AddTablePartitionContext): LogicalPlan = withOrigin(ctx) {
+    if (ctx.VIEW != null) throw new ParseException(s"Unsupported partitioned view", ctx)


I think @yhuai prefers to say something like Operation not allowed: partitioned views for the message (here and everywhere else)

Will do it now. Thanks!

andrewor14 · 2016-04-05T19:31:12Z

Looks pretty good, just a minor comment.

gatorsmile · 2016-04-05T22:28:53Z

retest this please

SparkQA · 2016-04-05T22:37:42Z

Test build #55031 has finished for PR 12169 at commit e5a8de7.

This patch fails to build.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- s\"its class is $
- s\"Unable to generate an encoder for inner class$`
- case class UnresolvedGenerator(name: String, children: Seq[Expression]) extends Generator
- case class UnresolvedDeserializer(deserializer: Expression, inputAttributes: Seq[Attribute])
- case class FunctionResource(resourceType: FunctionResourceType, uri: String)
- trait FunctionResourceLoader
- case class CatalogFunction(
- case class CreateFunction(
- case class DropFunction(
- case class StreamingRelation(dataSource: DataSource, sourceName: String, output: Seq[Attribute])
- case class StreamingExecutionRelation(source: Source, output: Seq[Attribute]) extends LeafNode

andrewor14 · 2016-04-05T23:42:00Z

Build is broken I think. retest this please

andrewor14 · 2016-04-05T23:42:40Z

sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/HiveSqlParser.scala

+   * }}}
   */
  override def visitCreateView(ctx: CreateViewContext): LogicalPlan = withOrigin(ctx) {
    // Pass a partitioned view on to hive.


this needs to be deleted

andrewor14 · 2016-04-05T23:43:00Z

LGTM we just need to make tests pass now

SparkQA · 2016-04-06T01:09:46Z

Test build #55049 has finished for PR 12169 at commit e5a8de7.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- s\"its class is $
- s\"Unable to generate an encoder for inner class$`
- case class UnresolvedGenerator(name: String, children: Seq[Expression]) extends Generator
- case class UnresolvedDeserializer(deserializer: Expression, inputAttributes: Seq[Attribute])
- case class FunctionResource(resourceType: FunctionResourceType, uri: String)
- trait FunctionResourceLoader
- case class CatalogFunction(
- case class CreateFunction(
- case class DropFunction(
- case class StreamingRelation(dataSource: DataSource, sourceName: String, output: Seq[Attribute])
- case class StreamingExecutionRelation(source: Source, output: Seq[Attribute]) extends LeafNode

SparkQA · 2016-04-06T03:32:00Z

Test build #55072 has finished for PR 12169 at commit 93e51b4.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

andrewor14 · 2016-04-06T05:33:11Z

Merged into master.

yhuai · 2016-04-06T06:17:48Z

@gatorsmile Seems scala 2.10 build is broken. Can you take a look?

gatorsmile · 2016-04-06T06:21:51Z

Sure. Could you show me how to find the build log?

gatorsmile · 2016-04-06T06:24:19Z

Or I need to do it in my local environment?

gatorsmile · 2016-04-06T06:26:58Z

https://amplab.cs.berkeley.edu/jenkins/job/spark-master-compile-maven-scala-2.10/ ?

gatorsmile · 2016-04-06T06:28:06Z

I see the problem.

[error] /home/jenkins/workspace/spark-master-compile-maven-scala-2.10/sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveDDLCommandSuite.scala:266: value contains is not a member of Option[String]
[error]     assert(desc.viewText.contains("SELECT * FROM tab1"))
[error]                          ^
[error] /home/jenkins/workspace/spark-master-compile-maven-scala-2.10/sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveDDLCommandSuite.scala:267: value contains is not a member of Option[String]
[error]     assert(desc.viewOriginalText.contains("SELECT * FROM tab1"))
[error]                                  ^
[error] /home/jenkins/workspace/spark-master-compile-maven-scala-2.10/sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveDDLCommandSuite.scala:293: value contains is not a member of Option[String]
[error]     assert(desc.viewText.contains("SELECT * FROM tab1"))
[error]                          ^
[error] /home/jenkins/workspace/spark-master-compile-maven-scala-2.10/sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveDDLCommandSuite.scala:294: value contains is not a member of Option[String]
[error]     assert(desc.viewOriginalText.contains("SELECT * FROM tab1"))
[error]                                  ^
[error] four errors found
[error] Compile failed at Apr 5, 2016 10:59:09 PM [10.502s]

gatorsmile · 2016-04-06T06:28:28Z

Will fix it using a PR with [HOT]. Sorry for it.

gatorsmile and others added 30 commits November 13, 2015 14:50

Merge remote-tracking branch 'upstream/master'

01e4cdf

Merge remote-tracking branch 'upstream/master'

6835704

Merge remote-tracking branch 'upstream/master'

9180687

SPARK-11633

b38a21e

Merge remote-tracking branch 'upstream/master' into joinMakeCopy

d2b84af

Merge remote-tracking branch 'upstream/master'

fda8025

Merge branch 'master' of https://github.com/gatorsmile/spark

ac0dccd

Merge remote-tracking branch 'upstream/master'

6e0018b

converge

0546772

converge

b37a64f

Merge remote-tracking branch 'upstream/master'

c2a872c

Merge remote-tracking branch 'upstream/master'

ab6dbd7

Merge remote-tracking branch 'upstream/master'

4276356

Merge remote-tracking branch 'upstream/master'

2dab708

Merge remote-tracking branch 'upstream/master'

0458770

Merge remote-tracking branch 'upstream/master'

1debdfa

Merge remote-tracking branch 'upstream/master'

763706d

Merge remote-tracking branch 'upstream/master'

4de6ec1

Merge remote-tracking branch 'upstream/master'

9422a4f

Merge remote-tracking branch 'upstream/master'

52bdf48

Merge remote-tracking branch 'upstream/master'

1e95df3

Merge remote-tracking branch 'upstream/master'

fab24cf

Merge remote-tracking branch 'upstream/master'

8b2e33b

Merge remote-tracking branch 'upstream/master'

2ee1876

Merge remote-tracking branch 'upstream/master'

b9f0090

Merge remote-tracking branch 'upstream/master'

ade6f7e

Merge remote-tracking branch 'upstream/master'

9fd63d2

Merge remote-tracking branch 'upstream/master'

5199d49

Merge remote-tracking branch 'upstream/master'

404214c

Merge remote-tracking branch 'upstream/master'

c001dd9

hvanhovell reviewed Apr 5, 2016
View reviewed changes

Merge remote-tracking branch 'upstream/master'

65bd090

gatorsmile added 3 commits April 5, 2016 06:13

address comments.

14df7ab

Merge remote-tracking branch 'upstream/master'

babf2da

Merge branch 'viewPartition' into viewPartitionNew

140f859

# Conflicts: # sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveDDLCommandSuite.scala

andrewor14 reviewed Apr 5, 2016
View reviewed changes

gatorsmile added 2 commits April 5, 2016 14:28

address comments

c9d40e1

Merge remote-tracking branch 'upstream/master' into viewPartitionNew

e5a8de7

andrewor14 reviewed Apr 5, 2016
View reviewed changes

address comments

93e51b4

asfgit closed this in 68be5b9 Apr 6, 2016

andrewor14 mentioned this pull request Apr 8, 2016

[SPARK-14132][SPARK-14133][SQL] Alter table partition DDLs #12220

Closed

Conversation

gatorsmile commented Apr 5, 2016

What changes were proposed in this pull request?

How was this patch tested?

Uh oh!

hvanhovell Apr 5, 2016

Choose a reason for hiding this comment

Uh oh!

gatorsmile Apr 5, 2016

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Apr 5, 2016

Uh oh!

SparkQA commented Apr 5, 2016

Uh oh!

gatorsmile commented Apr 5, 2016

Uh oh!

andrewor14 Apr 5, 2016

Choose a reason for hiding this comment

Uh oh!

gatorsmile Apr 5, 2016

Choose a reason for hiding this comment

Uh oh!

andrewor14 commented Apr 5, 2016

Uh oh!

gatorsmile commented Apr 5, 2016

Uh oh!

SparkQA commented Apr 5, 2016

Uh oh!

andrewor14 commented Apr 5, 2016

Uh oh!

andrewor14 Apr 5, 2016

Choose a reason for hiding this comment

Uh oh!

andrewor14 commented Apr 5, 2016

Uh oh!

SparkQA commented Apr 6, 2016

Uh oh!

SparkQA commented Apr 6, 2016

Uh oh!

andrewor14 commented Apr 6, 2016

Uh oh!

yhuai commented Apr 6, 2016

Uh oh!

gatorsmile commented Apr 6, 2016

Uh oh!

gatorsmile commented Apr 6, 2016

Uh oh!

gatorsmile commented Apr 6, 2016

Uh oh!

gatorsmile commented Apr 6, 2016

Uh oh!

gatorsmile commented Apr 6, 2016

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants